Skip to content

branch-4.1: [feature](statistics) Skip collecting stats for long string columns #62686#63303

Open
yujun777 wants to merge 1 commit into
apache:branch-4.1from
yujun777:cherry-pick-62686-to-branch-4.1
Open

branch-4.1: [feature](statistics) Skip collecting stats for long string columns #62686#63303
yujun777 wants to merge 1 commit into
apache:branch-4.1from
yujun777:cherry-pick-62686-to-branch-4.1

Conversation

@yujun777
Copy link
Copy Markdown
Contributor

cherry-pick: #62686

…pache#62686)

Issue Number: close #0

Problem Summary: When a string column contains extremely long values,
running
ANALYZE on it can consume excessive memory and may even OOM the BE. This
PR
introduces an inline per-row guard that short-circuits stats collection
for any
string column whose rows exceed
`Config.statistics_max_string_column_length`
(default 1024 bytes).

The guard is injected into every statistics-collection SQL template
(FULL_ANALYZE_TEMPLATE, LINEAR_ANALYZE_TEMPLATE, DUJ1_ANALYZE_TEMPLATE,
HMS full/sample) as a `NoneMovableFunction`:

assert_true(col IS NULL OR LENGTH(col) <= N,
'ANALYZE_SKIP_LONG_STRING_COLUMN')
    AS __lc

When any row violates the predicate, `assert_true` triggers a controlled
error
carrying the `ANALYZE_SKIP_LONG_STRING_COLUMN` marker.
`BaseAnalysisTask`
walks the exception cause chain for this marker and, when detected,
converts
the failure into a FINISHED task with a skip message so users see why
the
column was dropped (via SHOW ANALYZE and via OK-packet info for sync
analyze).
Other columns in the same job are unaffected.

Setting `statistics_max_string_column_length = 0` disables the guard
entirely
(upstream behavior).

Added FE config `statistics_max_string_column_length` (default 1024).
String
columns whose row byte length exceeds this value are skipped during
ANALYZE
with a visible skip message, instead of risking OOM. Set to 0 to
disable.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yujun777 yujun777 requested a review from yiguolei as a code owner May 15, 2026 11:11
@yujun777
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen
Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 74.60% (47/63) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants